ww 


'■N  TEST  CHART 

*NDA»DS  1%<  ft 


_ UNCLASS  I  ri£D _ 

set. O'"  -  I.ASS'Fti-.ON  3  c  '^is  Om  EkiiiX! 

REPORT  DOCUMENTATION  PAGE 


'  report  number  i  govt  accession  no 

AI  Memo  975 

4  TlT^E  t  snd  Subtltlt) 

Relaxing  the  Brightness  Constancy  Assumption 
in  Computing  Optical  Flow 


READ  INSTRUCTIONS 

_ BEFORE  COMPLETING  FORM 

i  recipient's  Catalog  number 


S  TYPE  O  F  AEAQAT  •  PERIOD  COVERED 


AI  Memo 

t  PERFORMING  OKS  REPORT  NUMBER 


|7  ajThO*'i; 


Shahriar  Negahdaripour  Michael  Gennert 


>  performing  organization  name  AND  AOORESS 
Artificial  Intelligence  Laboratory 
545  Technology  Square 
Cambridge,  MA  02139 

M  CONTROLLING  OFFICE  NAME  ANO  AOORESS 

Advanced  Research  Projects  Agency 
1400  Wilson  Blvd. 

Arlington,  VA  22209 

“  monitoring  agency  NAME  S  AOOREtyil  dlllmtmni  Irmm  Cmnttmlllnt  Olllca) 

Office  of  Naval  Research 
Information  Systems 
Arlington,  VA  22217 

_  OISTMiauTlON  statement  7*7  this  R •»**!;  — ■ 


I  CONTRACT  or  GRANT  NUMOERraj 

N00014-85-K-0124 

DACA76-85-C-0100 


10.  PROGRAM  ELEMENT  PROJECT  TASK 
AREA  A  WORK  UNIT  NUMBERS 


12.  REPORT  DATE 

June  1987 

IS.  NUMBER  or  PAGES 

24 

ii.  security  class  >•>  hh«  tmpmn) 
UNCLASSIFIED 

’  lla.  DECLASSIFICATION/ DOWNGRADING 
SCHEDULE 


Distribution  is  unlimited. 


17  DISTRIBUTION  STATEMENT  (ml  Ufa  akairpci  mmimrmd  In  Black  SB,  II  Wfffarani  Ham  Mm 


I*  SUPPLEMENTARY  NOTES 


[  If.  KEY  WORDS  rCanllnwa  an  tmmmrmm  alWa  II  nMfiiarr  CM  I  Wan  Ml*  Hr  kIPCk  —tirj 


DTIC 


NOV  0  6  1987  :i 


Optical  Flow,  Brightness  Constancy  Assumption,  Passive  Navigation/ 
Smoothness  Constraint 


I  20  ABSTRACT  (C •***"*•  m  r«v«rc«  It  m*4  Id** tiff  If  Mtd  whr) 

'  Optical  flow  is  the  apparent  (or  perceived)  motion  of  image  brightness  patterns 
arising  from  relative  motion  of  objects  and  observer.  Estimation  of  the  optical 
flow  requires  the  application  of  two  kinds  of  constraint:  the  flew  field  smoothne 
constraint  and  the  brightness  constancy  constraint.  The  brightness  constancy 
constraint  permits  one  to  match  image  brightness  values  across  images,  but  is 
very  restrictive.  We  propose  replacing  this  constraint  with  a  more  general  const¬ 
raint,  which  permits  a  linear  transformation  between  image  brightness  values. 

t  inexact 


m  a  V:  1  «T:  Tm\ 


DO  ,  1473  ITiON  OF  I  NOV  (1  It  OBSOLETE 

S/N  0  :0  >•  0  !«•  ««0I  1 


UNCLASSIFIED 

I E  Ciip’Yy  CL  AttlPICATlOW  «5F_Tm<«  P  AO  t FpMPn 


Block  20  cont. 


matching  is  allowed.  W^e  describe  the  implementation  on  a  highly  parallel 
computer,  -and-present  sample  results. 


/" 


U' 


MASSACHUSETTS  INSTITUTE  OF  TECHNOLOGY 
ARTIFICIAL  INTELLIGENCE  LABORATORY 


A.I.  Memo  No.  975 


June,  1987 


Relaxing  the  Brightness  Constancy  Assumption  in  Computing  Optical  Flow 


Michael  A.  Gennert  and  Shahriar  Negahdaripour 


Abstract:  Optical  flow  is  the  apparent  (or  perceived)  motion  of  image  brightness  patterns 
arising  from  relative  motion  of  objects  and  observer.  Estimation  of  the  optical  flow  requires 
the  application  of  two  kinds  of  constraint:  the  flow  field  smoothness  constraint  and  the 
brightness  constancy  constraint.  The  brightness  constancy  constraint  permits  one  to  match 
image  brightness  values  across  images,  but  is  very  restrictive.  We  propose  replacing  this 
constraint  with  a  more  general  constraint,  which  permits  a  linear  transformation  between 
image  brightness  values.  The  transformation  parameters  are  allowed  to  vary  smoothly,  so 
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computer,  and  present  sample  results. 
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Abstract:  Optical  flow  is  the  apparent  (or  perceived)  motion  of  image  brightness  pat¬ 
terns  arising  from  relative  motion  of  objects  and  observer.  Estimation  of  the  optical  flow 
requires  the  application  of  two  kinds  of  constraint:  the  flow  field  smoothness  constraint 
and  the  brightness  constancy  constraint.  The  brightness  constancy  constraint  permits 
one  to  match  image  brightness  values  across  images,  but  is  very  restrictive.  We  propose 
replacing  this  constraint  with  a  more  general  constraint,  which  permits  a  linear  transfor¬ 
mation  between  image  brightness  values.  The  transformation  parameters  are  allowed  to 
vary  smoothly,  so  that  inexact  matching  is  allowed.  We  describe  the  implementation  on 
a  highly  parallel  computer,  and  present  sample  results. 

1  Introduction 

Optical  flow  is  the  apparent  (or  perceived)  motion  of  image  brightness  patterns  arising 
from  relative  motion  of  objects  and  observer.  Optical  flow  can  give  important  information 
about  motion  of  the  observer  (i.e.  passive  navigation),  motion  of  objects  in  the  scene,  and 
the  spatial  arrangement  of  these  objects.  Additionally,  discontinuities  in  the  optical  flow 
field  can  be  used  to  segment  the  image  into  regions  corresponding  to  different  objects. 

To  be  precise,  an  optical  flow  field  is  a  two-dimensional  vector  field  relating  brightness 
patterns  in  an  image  at  one  instant  of  time  to  brightness  patterns  at  the  next  instant  of 
time.  There  does  not  exist  a  unique  optical  flow  field  for  a  given  image  sequence;  rather 
there  are  infinitely  many  flow  fields  satisfying  the  image  constraints.  This  illustrates  one 
of  the  difficulties  associated  with  determining  optical  flow,  namely,  identifying  sufficient 
constraint  to  produce  a  unique  optical  flow  field.  The  other  difficulty  in  determining 
optical  flow  is  more  fundamental,  and  involves  finding  image  elements  to  be  placed  into 
correspondence. 

As  in  stereo,  methods  for  computing  optical  flow  can  be  classified  according  to  whether 
detected  features  are  used  as  primitive  elements,  or  whether  image  brightness  values  (and 
gradients)  are  used  directly.  Feature-based  approaches  to  optical  flow  use  detected  edges 
almost  exclusively  (Hildreth  [1983],  Davis  et  al  [1983],  Murray  As  Buxton  [1984],  Wohn 
[1984]).  Although  it  is  possible  to  use  detected  points  to  process  visual  motion,  no  one 
seems  to  have  attempted  to  determine  optical  flow  from  isolated  point  displacements. 
Horn  At  Schunck  [1981]  were  among  the  first  to  use  image  brightness  directly  to  determine 
optical  flow.  They  solved  the  problems  mentioned  above  by  identifying  two  constraints: 
the  spatial  smoothness  constraint  and  the  brightness  constancy  constraint.  Other  methods 
based  on  various  smoothness  assumptions  have  been  proposed  (Prager  At  Arbib  [1983], 
Paquin  As  Dubois  [1983],  Yashida  [1983],  Anandan  [1984]). 

The  smoothness  constraint  arises  from  the  observation  that  most  visual  motion  is 
the  result  of  objects  of  finite  size  undergoing  rigid  motion  or  deformation.  Neighboring 
object  points  have  similar  motions  or  velocities,  and  to  the  extent  that  they  project 
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to  neighboring  image  points,  neighboring  image  points  will  also  have  similar  motions. 
Therefore,  the  optical  flow  field  should  be  smooth  almost  everywhere.  Exceptions  occur 
at  occluding  boundaries,  where  neighboring  image  points  are  not  generally  the  projections 
of  neighboring  object  points. 

The  brightness  constancy  constraint  rests  on  the  assumption  that  the  brightness  of 
a  small  image  patch  remains  approximately  constant  as  the  corresponding  surface  patch 
moves  in  the  environment.  This  is  a  reasonable  assumption  when  the  lighting  conditions 
are  unchanged  between  successive  images,  object  surfaces  are  non-specular,  and  there  is 
only  a  small  amount  of  motion  between  image  frames.  If  these  conditions  are  met,  then 
the  brightness  constancy  constraint  will  apply  approximately  at  all  image  points. 

It  should  be  noted  that  the  smoothness  constraint  depends  on  the  scene  structure, 
and  is  independent  of  illumination,  surface  reflectance  characteristics,  and  the  type  and 
degree  of  motion  involved.  On  the  other  hand,  the  brightness  constancy  constraint  does 
not  depend  on  scene  structure  (except  for  the  influence  of  surface  microstructure  on 
reflectance),  but  instead  depends  on  the  degree  and  types  of  motion,  and  factors  such 
as  illumination  and  surface  reflectance  which  affect  image  irradiance.  Violations  of  the 
brightness  constancy  constraint,  when  they  occur,  affect  image  patches  or  even  entire 
images.  Therefore,  it  is  important  to  find  ways  to  relax  this  constraint. 

Cornelius  &  Kanade  [1983]  propose  a  variation  of  the  Horn  &  Schunck  [1981]  method. 
In  their  formulation,  they  allow  gradual  changes  in  the  way  an  object  appears  in  a  se¬ 
quence  of  images.  An  image  point  does  not  have  to  preserve  the  same  brightness  value 
as  the  object  point  that  give  rise  to  it  moves  in  the  environment,  however,  the  variation 
is  enforced  to  be  smooth  from  one  image  point  to  the  next. 

In  this  paper,  we  propose  a  new  formulation  by  relaxing  the  brightness  constancy  con¬ 
straint.  Our  approach  does  not  require  exact  brightness  matching  across  image  frames, 
but  accepts  even  approximate  matches.  We  achieve  this  by  permitting  a  linear  transfor¬ 
mation  of  image  brightness  values  between  image  frames,  and  constraining  the  allowed 
transformations.  Our  formulation,  in  special  cases,  reduces  to  that  of  Cornelius  &  Kanade 
[lQ8  -  or  that  of  Horn  &  Schunck  [1981]. 

2  Mathematics  of  Optical  Flow 

Lot  coordinate  system  be  aligned  with  the  imaging  system  so  that  the  2-axis  points 
aio '  he  optical  axis.  The  image  plane  can  arbitrarily  be  chosen  to  lie  at  z  =  1  so  that 
in.ug  points  are  given  by  r  =  (x,  y,  l)r.  Let  E( r,  t)  denote  the  brightness  of  image  point 
r  at  i  hue  t.  At  a  later  time  t  +  St ,  the  brightness  pattern  at  r  will  have  moved  to  a  new 
1<o,i  on  r  f  <5  r  =  (x  +  6xy  y  +  6y,  l)7’.  The  optical  flow  is  the  velocity  field  arising  from 
th  drived  motion  of  image  points  r.  It  is  derived  from  the  displacements  of  image 
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points  by  taking  the  limit  as  6*  — ►  0, 

dr  .dx  dy  .T  . 

I,  =  Tt=(S'7t'a)  =(“'“'°)  ■ 

where  u  and  v  denote  the  components  of  optical  flow  r*. 

The  brightness  constancy  constraint  of  Horn  &  Schunck  [1981]  expresses  the  restriction 
that  the  brightness  of  an  image  patch  remains  approximately  constant  as  the  surface  patch 
that  gives  rise  to  that  image  patch  moves  in  the  environment.  Setting  the  total  derivative 
of  image  brightness  equal  to  zero,  we  can  write 


dE 

dt 


=  0. 


Applying  the  chain  rule,  we  obtain 

dE 

dt 


dE  dx  dE  dy_ 
dx  dt  dy  dt  ’ 


or 


Et  +  Et  •  rt  =  0, 

where  E,  =  (dE /dx,dE /dy, 0)r.  This  equation  is  sometimes  referred  to  as  the  image 
brightness  change  constraint  equation  derived  under  the  constant  brightness  assumption. 
It  has  also  been  referred  to  as  the  optical  flow  constraint  equation. 

Assuming  brightness  constancy,  we  define  optical  flow  as  any  2D  vector  field  tt,  defined 
on  the  image  plane,  that  satisfies  the  image  brightness  continuity  equation. 

2.1  Image  Brightness  Constraint 

Nagel  [1983a, 1983b]  suggests  a  formulation  that  incorporates  second-order  effects  in  or¬ 
der  to  obtain  a  better  estimate  of  the  optical  flow  around  edges  and  corners.  In  his 
formulation,  the  brightness  change  constraint  equation  is  written 

1 


Et  +  Er  •  rt  +  -rt  E„rt  =  0, 


where 


/  d2E/dx 2  d2E/dxdy  O' 

Err  =  d2E/dxdy  d2E/dy2  0 
VO  0  0. 

Here,  again,  the  constraint  equation  rests  on  the  brightness  constancy  assumption. 

The  formulation  proposed  by  Cornelius  &  Kanade  [1983]  allows  gradual  changes  in 
the  way  an  object  appears  in  a  sequence  of  images.  In  their  formulation,  the  brightness 
change  constraint  is  written 

dE 

W  =  E,  +  K.r,. 
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An  image  point  does  not  have  to  preserve  the  same  brightness  value  as  the  object  point 
that  gives  rise  to  it  moves  in  the  environment.  Hence,  the  rate  of  brightness  change  can 
be  non-zero;  that  is, 

dE 

More  generally,  we  propose  a  formulation  that  allows  a  linear  transformation  between 
brightness  values  in  consecutive  images.  We  choose  a  linear  transformation  because  it  is 
one  of  the  simplest  non-trivial  transformations.  This  is  a  less  restrictive  assumption  than 
brightness  constancy,  and  can  be  formulated  as 

E( r  +  Sr,  t  +  St )  =  M(r,  t)  E{ r,  t )  +  0( r,  t), 

where  M  is  the  multiplier  and  C  is  the  offset  functions  in  the  linear  transformation.  This 
is  our  revised  image  brightness  change  constraint  equation. 

For  small  St,  we  expect  M  to  be  close  to  1,  and  C  to  be  close  to  0.  Since  we  are 
dealing  with  incremental  changes,  we  can  let  Af  =  1  +  6m  and  C  =  Sc.  In  fact,  m  and  c 
are  the  quantities  of  interest  to  us.  Noting  that  Sm  —*  0  and  Sc  — ►  0  as  St  — ►  0,  we  can 
define  time  derivatives,  and  c*, 

m*  =  lim  and  ct  =  lim 

6t—0  St  St-* 0  St 

that  we  will  use  in  our  derivation. 

Rewriting  the  brightness  change  constraint  equation,  we  obtain 

E(r  +  6r,t  +  St)  =  (l  +  6m(r,  t)]  E{r,  t)  +  Sc( r,  t). 

The  left  hand  side  can  be  expanded  as  follows: 

dE  dE 

E{ r  4-  Sr,  t  +  St)  =  E{ r,  t)  -1-  —  •  St  +  — -St  +  0(e)  =  Er  •  Sr  +  Et6t  +  O(e). 

oT  vt 

Substituting  this  into  the  constraint  equation  and  simplifying,  we  have 

Er  •  Sr  +  EtSt  —  E  Sm  —  Sc  +  0(e)  =  0. 

Finally,  dividing  through  by  St  and  taking  the  limit  as  St  — ►  0,  we  arrive  at 

Et  ■+■  Et  •  Tt  —  E  mt  —  ct  —  0. 

This  is  our  revised  optical  flow  constraint  equation.  In  the  special  case  that  M  =  1  and 
C  t  0  (and,  hence,  mt  —  0  but  c<  ^  0),  this  becomes  similar  to  the  constraint  equation 
in  the  Cornelius  Si  Kanade  [1983]  formulation: 

ct  =  Et  4  Er  •  r(. 

Further,  in  the  more  restricted  case  that  M  =  1  and  0  =  0  (that  is,  mt  =  c<  =  0),  our 
constraint  equation  reduces  to  the  one  in  the  Horn  Si  Schunck  [1981]  formulation: 

Et  +  Er-  rt  =  0. 


m 
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Figure  1.  Corresponding  iso-brightness  contours  in  an  image  sequence. 


3  The  Aperture  Problem 


There  are  an  infinite  number  of  valid  optical  flows  given  an  image  sequence.  To  see 
this,  we  note  that  if  r*  is  an  optical  flow,  so  is 


r't  =  r t  +  /(x,y)  ( E ,  x  z) 


for  any  f(x,  y),  where  z  is  a  unit  vector  perpendicular  to  the  image  plane.  First,  note 
that  rj  •  z  =  rt  •  £  =  0,  as  it  should.  Furthermore,  we  have 


Et  +  E,-  [rt  +  /  ( Er  x  £)]  -  E  mt  -  ct  =  Et  +  E,  ■  rt  -  E mt  -  ct  =  0. 


This  can  also  be  explained  graphically.  Consider  the  simple  case  where  M  =  1  and  C  =  0, 
so  that  the  constraint  equation  reduces  to  that  of  Horn  &  Schunck  formulation: 


Er  •  Tt  +  E(  =  0. 


Referring  to  figure  1,  suppose  that  C'  is  a  contour  of  constant  brightness  in  the  second 
image  corresponding  to  contour  C  in  the  first  image.  It  is  not  easy  to  decide  which  point 
P'  on  C'  corresponds  to  a  particular  point  P  on  C  since  the  contour  generally  changes 
shape  as  the  object  moves  in  the  environment  (Horn  [1986]) .  In  fact,  there  are  many 
possible  ways  to  establish  correspondence  between  points  on  contours  C  and  C' .  This 
ambiguity  has  been  referred  to  as  the  aperture  problem.  In  terms  of  the  iso-brightness 
contours,  any  vector  field  that  transforms  contour  C  into  contour  C1  is  an  acceptable 
optical  flow. 
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In  our  extended  formulation,  the  indeterminacy  of  the  optical  flow  is  even  worse  than 
this  example  suggests.  Since  mj  and  c*  are  unconstrained,  the  optical  flow  field  can  be 
completely  arbitrary,  with  either  or  both  of  these  transformation  fields  varying  in  such  a 
way  as  to  guarantee  that  the  brightness  constraint  is  obeyed.  It  is,  therefore,  necessary 
to  select,  out  of  the  infinite  number  of  possible  optical  flows,  one  which  is  consistent  with 
the  physical  constraints  of  the  problem.  One  may  hope  to  obtain  an  optical  flow  field 
that  approximates  the  apparent  motion  of  brightness  patterns  in  the  image  as  judged  by 
a  human  observer. 

3.1  Smoothness  Assumption 

Discontinuities  in  depth  (for  example,  at  occluding  boundaries)  give  rise  to  disconti¬ 
nuities  in  the  optical  flow  field.  Also,  object  motions  may  be  different  across  occluding 
boundaries,  which  can  give  rise  to  discontinuities  in  the  optical  flow.  Additionally,  discon¬ 
tinuities  can  be  expected  in  mt  and  c*,  if  illumination  conditions  or  reflectance  properties 
that  depend  on  surface  material  change  abruptly  as  the  surface  moves  in  the  environment. 
In  the  absence  of  depth  discontinuities  or  abrupt  changes  in  illumination  or  surface  re¬ 
flectance  properties,  the  optical  flow  and  transformation  fields  are  expected  to  be  smooth. 
Based  on  these  facts,  we  require  that  the  optical  flow,  the  multiplier,  and  the  off-set  fields 
should  be  consistent  with  our  revised  optical  flow  constraint  equation,  and  should  vary 
smoothly  from  one  image  point  to  the  next. 

Smoothness  can  be  imposed  by  minimizing  a  functional  that  is  a  measure  of  departure 
from  smoothness.  Horn  &  Schunck  [1981]  proposed  minimizing  the  integral  of  the  square 
of  the  magnitude  of  the  gradient  of  the  optical  flow.  Hildreth  [1983]  investigated  a  similar 
formulation,  but  incorporated  different  measures  of  smoothness. 

The  gradient  of  the  optical  flow  is 

dr  ( du/dy  0\ 

Vr t  —  -rr~  =  [  dv/dx  dv/dy  0  I  . 

3r  \  0  0  0  / 

The  measure  of  departure  from  smoothness  that  is  to  be  minimized  is  written 

e*  =  J  J  \\VrtW\dxdy. 

Ilea,  ;  •  ]J*  denotes  the  Euclidean  or  Frobenius  norm  of  a  matrix,  which  is  the  sum  of  the 
square  of  all  the  elements  of  the  matrix. 

Similarly,  smoothness  deviations  can  be  defined  for  the  transformation  fields 

em  =  j  j  \\^mt\\]dxdy  and  ec  J  J  j|  Vct||J  dx  dy. 
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The  image  brightness  constraint  a  id  the  smoothness  constraints  can  be  combined  by 
defining  a  single  functional  that  weighs  each  contribution.  Rather  than  enforcing  the 
brightness  change  constraint  exactly,  we  use  a  penalty  term  that  measures  the  square  of 
the  error  in  the  constraint  equation  over  the  whole  image: 


To  ensure  that  the  optical  flow  and  the  transformation  fields  (approximately)  satisfy  the 
optical  flow  constraint  equation,  we  want  ej  to  be  small. 

All  together,  the  problem  can  be  formulated  as  that  of  minimizing  the  functional 

e  =  ej,  +  \aea  +  A  mem  +  Xcec, 

where  A,,  Am,  Ac  weigh  the  total  error  contributed  by  each  term. 

Using  variational  calculus,  the  Euler-Lagrange  equations  for  this  problem  can  be 
found.  These  equations  form  a  set  of  necessary  conditions  that  a  solution  to  our  mini¬ 
mization  problem  has  to  satisfy.  Sufficiency  is  not  guaranteed,  in  particular,  it  is  possible 
for  a  particular  proposed  solution  to  obey  the  Euler-Lagrange  equations  yet  not  be  a 
global  minimum.  This  will  occur  at  local  minima,  points  of  inflection,  and  local  maxima. 
Note,  however,  that  there  is  no  global  maximum,  as  the  functional  is  not  bounded  from 
above.  We  will  not  address  the  question  of  sufficiency  further  in  this  paper. 

The  variational  problem  is  solved  by  using  the  formula 

*'  -  h*'-  -  =  °- 

where  V  is  the  integrand  in  the  cost  functional  and  f  is  each  of  u,v,m*  or  c*,  in  turn. 
Applying  the  above  formula,  we  obtain 

E 

V2u  =  — -(Et  +  Exu  +  Eyv  -  Emt  -  ct), 

A 8 

E 

V2v  =  -^-{Et  4-  Exu  +  Eyv  -Emt-  ct), 

— E 

V2m(  -  — — (Et  +  Exu  +  Eyv  -  Emt  —  c*), 

Am 

V2ct  =  —  (Et  +  Exu  +  Eyv  -  E  mt  -  ct). 

Ac 

For  a  well-posed  problem,  we  need  to  specify  the  appropriate  boundary  conditions.  In  the 
absence  of  fixed  boundary  conditions  (values  of  u,  v,m*,  and  c<  on  the  image  boundaries), 


V 


l*. 


It 
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we  need  to  specify  natural  boundary  conditions.  For  our  problem,  the  natural  boundary 
condition  is 

{fx,fy)T-  »  =  0, 


where  n  is  a  unit  vector  perpendicular  to  the  boundary.  Again,  /  can  be  any  one  of 
u,v,mt  or  ct,  in  turn. 


4.1  A  Discrete  Implementation 


In  the  discrete  domain,  the  Laplacian  operator  V2  can  be  approximated  as  a  center- 
surround  operator 

v2/  »  k(7-  /), 


where  /  (the  “surround”)  is  an  averaged  or  smoothed  version  of  /.  For  example,  we  can 
use  the  following  simple  approximation: 


/  —  jifi+lj  +  /»'-!,>  +  /t,j  +  l  +  /t,jr  —  1 )  - 


The  scale  factor  k  can  be  absorbed  into  the  appropriate  A  and,  therefore,  need  not  be 
considered  further. 

Substituting  the  approximation  to  the  Laplacian  in  the  Euler-Lagrange  equation, 
derived  earlier,  we  can  write  a  single  matrix  equation 


Af  =  g(f), 


where 


f  - 


and 


/  u  \ 

/  ■  E x 

\ 

mt 

II 

Xsv  -  EyEt 
XmrfTt  +  EEt 

\ct) 

V  Accf  +  Et 

J 

El  +  A 

3  Ex  Ey 

-EXE 

Ex 

ExEy 

El  +  A, 

-EyE 

Ey 

-EXE 

'  -EyE 

E2  +  \m 

E 

-Ex 

~Ey 

E  1 

1-  A 

These  equations  have  to  be  solved  iteratively  since  the  optical  flow,  at  each  image  cell., 
depends  on  the  average  of  the  optical  flow  from  the  neighboring  cells.  The  same  is  true 
for  mt  and  c*. 

Solving  for  the  unknown  fields,  u,v,mt ,  and  c<,  we  find  that 


f  =  A-'«(f), 


w 


where 


AcAmA#  +  A„,A,+  _  Ip  Jp  \  \  TP  tp  \  \ 

&\c\.  +  E*\c\m  rJxCjyAcAm  L,xUAcAt 

—  JP  JP  \  \  AcAm  A,+AmA,+  IP  IP  \  \ 

£jx£jyAcAm  E2XeX.+EjXcXm  ^y^^c^s 

ExEXcXa  EyEXcXa  (Ei+E}+x.)xcx. 


Ex  XmXa 


EyXmXg 


ExXmXe 
E  y  X  jn  X  a 


E2X7+ 

(Ej+E2+X.)XmX. 


a  —  XmXa  +  E  AcAa  +  (El  +  Ey  4-  Aa)AcAmAa. 

This  is  a  system  of  linear  equations,  which  can  be  used  to  recover  the  optical  flow  u  and 
v,  and  the  transformation  fields  mj  and  c*. 

The  Held  equations  can  be  solved  iteratively,  at  every  image  cell,  according  to  the 
equation 


rt  +  l  _ 


A-^f), 


where  k  is  the  iteration  number.  The  matrix  A  (or  A-1)  depends  only  on  the  observed 
data.  It  needs  to  be  computed  once,  but  it  differs  from  point  to  point. 

This  formulation,  in  general,  requires  a  lot  of  computation  and  is  not  really  suitable 
for  implementation  on  a  serial  machine.  It  can,  however,  be  readily  implemented  on  a 
highly  parallel  computer,  such  as  the  Connection  Machine™.  For  a  128  by  128  image,  the 
Connection  Machine  implementation  runs  approximately  1000  faster  than  a  Symbolics 
3640™  Lisp  Machine  implementation. 


5  Examples 


Example  1  —  Multiplier  Effect  with  No  Offset:  Figure  2  shows  a  pair  of  image 
frames  from  a  synthetic  motion  sequence.  Each  image  contains  a  background  texture 
and  a  central  texture;  each  texture  is  gaussian-smoothed  uniform  noise.  Sharp  texture 
boundaries  between  the  background  and  the  central  circular  object  have  been  preserved 
in  each  image.  The  circular  region  undergoes  rigid  rotation  between  frames.  In  addition 
to  the  rotation  of  the  central  circular  region,  the  second  image  has  been  multiplied  by 
a  factor  which  varies  linearly  from  0.75  in  the  lower  left  corner  to  1.25  in  the  upper 
right  corner,  as  shown  in  Figure  3a.  An  offset  of  zero  was  used  and  all  A  parameters 
were  set  to  1.  The  computed  multiplier  held,  shown  in  Figure  3b,  varies  in  a  range  from 
0.82  in  the  lower  left  corner  to  1.22  in  the  upper  right  corner.  Here  the  linear  trend  is 
clearly  discernible.  The  computed  optical  flow  after  100  iterations  is  shown  in  Figure 
4  (the  needles  indicate  flow  direction  and  rate).  The  offset  field  for  this  example  had 
negligible  values,  with  absolute  values  not  exceeding  0.002.  This  is  to  be  expected,  as  the 
experiment  was  designed  so  that  the  offset  field  would  not  be  needed. 
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Figure  4.  Example  1:  The  computed  optical  flow  using  the  method  described  in  this  paper  (all  A  param 
eters  were  set  equal  to  1). 
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Figure  5.  Example  1:  The  computed  optical  flow  using  the  method  described  in  this  paper  (Ap  =  0.1, 
but  the  remaining  A  parameters  were  set  equal  to  1). 
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More  accurate  results  were  obtained  for  Xp  —  0.1  (all  other  A  parameters  were  set  to  1) 
as  shown  in  figures  5.  The  off-set  field  was  negligible  with  absolute  values  not  exceeding 
0.0002,  and  the  multiplier  field  varied  in  a  range  from  0.76  in  the  lower  left  corner  to  1.26 
in  the  upper  right  corner. 

To  check  the  improvement  offered  by  this  algorithm,  the  Horn  k  Schunck  algorithm 
was  also  used  on  this  image  sequence.  This  is  equivalent  to  using  Am  =  Ac  =  oo  in  our 
formulation.  The  results  of  the  Horn  k  Schunck  algorithm,  Figure  6,  were  in  agreement 
near  the  center  of  the  image  and  the  upper  left  and  lower  right  corners,  where  the  multi¬ 
plier  was  approximately  1.  The  two  algorithms  did  not  agree,  and  the  unmodified  Horn 
&Schunck  algorithm  was  clearly  incorrect,  at  the  lower  left  and  upper  right  image  corners 
where  the  multiplier  had  its  greatest  effect.  This  illustrates  the  inability  of  the  Horn  k 
Schunck  algorithm  to  correctly  handle  images  sequences  where  the  brightness  constancy 
constraint  does  not  apply. 

Figure  7  shows  the  solution  obtained  using  the  algorithm  of  Cornelius  k  Kanade. 
(This  was  done  using  Am  =  oo  and  Ae  =  1  in  our  formulation.)  As  expected,  there  is 
not  much  improvement  over  the  solution  from  Horn  k  Schunck  algorithm  since  their 
algorithm  is  designed  to  compensate  for  effects  similar  to  an  offset  in  an  image  sequence 
(however,  the  offset  was  set  to  zero  for  this  example). 

Example  2  -  Multiplier  and  Offset  Effects:  Figure  8  shows  the  pair  of  images  for 
this  example.  The  motion  is  as  in  the  previous  case,  the  multiplier  field  varies  linearly 
from  0.9  in  the  lower  left  corner  to  1.1  in  the  upper  right  corner,  and  the  offset  is  5  (the 
grey-level  values  were  increased  by  5  units).  The  computed  optical  flow  using  our  method 
is  shown  in  Figure  9  (all  A  parameters  were  set  equal  to  1).  Figures  10  and  11  show  the 
same  using  the  methods  of  Horn  k  Schunck  and  Cornelius  k  Kanade,  respectively.  Again, 
these  were  obtained  by  setting  Am  =  Ac  =  oo  (to  simulate  Horn  k  Schunck  algorithm), 
and  Am  =  oo  and  Ac  =  1  (to  simulate  Cornelius  &  Kanade  algorithm)  in  our  formulation. 
These  results  are  reasonable  where  the  multiplier  and  offset  effects  approximately  cancel 
each  other.  They  break  down  where  the  multiplier  and  offset  have  their  greatest  effects; 
that  is,  in  the  upper  right  corner  of  the  image.  We  conclude  that  the  Horn  k  Schunck 
algorithm  does  not  correctly  handle  images  sequences  where  the  brightness  constancy 
constraint  is  violated.  Similarly,  the  algorithm  of  Cornelius  k  Kanade  breaks  down  in 
regions  of  the  image  where  multiplier  effects  are  dominant. 


6  Reducing  the  Computation 


In  many  practical  situations,  the  off-set  c*  is  near  0.  Therefore,  the  algorithm  can  be 


17 


;w$; 

<'■ 


Figures.  A  pair  of  image*  fr<  ••  «  ■  i  *  ?i<-t  j-  t*i-  t  ;•  e  ••  <j«n  me:  th*  circular  region  undergoes  rigid 

rotation,  the  multiplier  fit-M  v  r  v  !'•  *i  i'i  >  !i"  '"it  '  >•«■:  t<>  11  the  upper  right  corner, 

anil  the  offset  is  5  units.  _ 


18 


Computing  Optical  Flow 


Figure  9.  Example  2:  The  computed  optical  flow  using  the  method  described  in  this  paper. 


sped  up  by  ignoring  the  offset  term.  We  can  instead  use  the  updating  equation 


f'*+1  =  (A')”1  g'(f‘) 

to  compute  the  optical  flow  and  the  multiplier  fields.  Here,  we  have  defined 

f  u  \  (  Xau  —  ExEt  \ 

f'=  t,  ,  g'(f)=  Xsv-  EyEt  , 

\mt)  V  xrnmt  +  EEt  J 

and 

!  [El\m  +  E2\,  +  \m\,  -ExEyXm  EXE\S  \ 

(A')"1  =  -  —ExEy\m  E2 \m  +  E2Xt  +  XmXe  EyEXs 

a  V  ExEX t  EyEXt  E\Xt  +  Efrs  +  X]  J 

where 

a'  =  XmX]  +  E*X]  +  {El  +  El)XmX„ 

If  the  offset  term  is  not  negligible,  then  the  estimates  obtained  from  the  above  vector 
equation  may  be  used  as  initial  conditions  in  the  original  updating  equations. 

7  Summary  and  Extensions 

Much  of  the  existing  methods  for  computing  the  local  optical  flow  depend  on  two  kinds  of 
constraint:  the  flow  field  smoothness  constraint  and  the  brightness  constancy  constraint. 
The  brightness  constancy  constraint  permits  one  to  match  image  brightness  values  across 
images.  This  constraint  is  sometimes  very  restrictive. 

We  have  proposed  a  new  formulation  by  replacing  the  brightness  constancy  constraint 
with  a  more  general  constraint,  which  permits  a  linear  transformation  between  image 
brightness  values.  The  transformation  parameters  are  allowed  to  vary  slowly  in  space, 
so  that  inexact  matching  is  allowed.  We  have  formulated  the  problem  of  computing  the 
optical  flow  as  a  minimization  of  a  quadratic  cost  functional.  Using  variational  methods, 
we  have  shown  that  the  problem  reduces  to  solving  Laplacian  equations  for  the  two 
components  of  the  optical  flow  field  and  the  two  transformation  fields.  We  have  described 
the  implementation  on  a  highly  parallel  computer,  and  presented  sample  results. 

One  of  the  disadvantages  associated  with  the  use  of  the  smoothness  constraint  is 
the  degree  of  smoothness  imposed  on  the  unknown  velocity  and  transformation  fields. 
In  fact,  the  algorithm  developed  here  tends  to  smooth  over  discontinuities  both  in  the 
optical  flow  and  transformation  fields.  One  way  to  overcome  this  shortcoming  is  to  use 
“line  processes”  (Marroquin  [1984]).  Simply  put,  the  idea  is  to  incorporate  in  the  penalty 
function  the  cost  of  introducing  a  discontinuity  in  the  optical  flow  or  transformation  fields 
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instead  of  interpolating  smoothly  between  two  neighboring  points  when  the  gradient 
becomes  large.  A  line  process  is  a  boolean  variable;  it  takes  on  the  value  1  where  there 
is  a  discontinuity  and  0,  otherwise. 

There  is,  however,  a  drawback  associated  with  the  introduction  of  line  processes  in 
the  minimization  scheme,  namely,  that  the  cost  functional  becomes  non-convex.  This 
generally  calls  for  inventing  sophisticated  optimization  methods  that  can  be  computa¬ 
tionally  exhaustive;  for  example,  a  simulated  annealing  scheme  (Marroquin  [1984’).  or 
an  algorithm  based  on  neuronal  network  models  (Hopfield  &  Tank  [1985;,  Koch  et  ai. 
[1986*). 

Alternatively,  we  can  employ  the  graduated  non-convexity  algorithm  based  on  the 
concept  of  weak  continuity  constraints  (Blake  At  Zisserman[l986[).  Here,  the  process  of 
minimizing  a  non-convex  cost  functional  is  replaced  by  minimizing  a  sequence  of  cost 
functions,  the  first  of  which  is  a  convex  approximation  to  the  true  cost  functional  and  the 
last  one  is  the  true  non-convex  cost  functional.  Needless  to  say,  we  have  yet  to  implement 
any  of  these  schemes. 
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